1. Field of the Invention
Systems and methods consistent with the principles of the invention relate generally to information searching and, more particularly, to determining the similarity of queries by comparing the query distributions of those queries.
2. Description of Related Art
Existing information searching systems use search queries to search through aggregated data to retrieve specific information that corresponds to the received search queries. Such information searching systems may search information stored locally, or in distributed locations. The World Wide Web (“web”) is one example of information stored in distributed locations. The web contains a vast amount of information, but locating a desired portion of that information can be challenging. This problem is compounded because the amount of information on the web and the number of new users inexperienced at web searching are growing rapidly.
Search engines attempt to return hyperlinks to web documents in which a user is interested. Generally, search engines base their determination of the user's interest on search terms (called a search query) entered by the user. The goal of the search engine is to provide links to high quality, relevant results to the user based on the search query. Typically, the search engine accomplishes this by matching the terms in the search query to a corpus of pre-stored web documents. Web documents that contain the user's search terms are “hits” and are returned to the user.
Many users of a hypertext medium, such as the web, can read documents in more than one language. Consider, for example, a query in English from a user that can read English and Spanish. A conventional technique for identifying documents in Spanish for this English query involves translating the query to Spanish and then processing the translated query to identify matching Spanish documents.
Query terms are inherently ambiguous. Therefore, translating them is challenging. Some conventional approaches use a bilingual dictionary to perform query translations. It has been found, however, that using a bilingual dictionary results in noisy translations. The noisy translations may be due to many factors. For example, a translation may result in extraneous terms being added to the query because a dictionary entry may list several senses for a term. In other words, each term may have one or more possible translations in the dictionary. Also, general dictionaries often do not include technical terminology. This makes translation of technical query terms difficult.
Other conventional approaches rely either on “parallel corpora” (i.e., collections of documents in which each of the documents appears in two different languages) or “co-occurrence statistics” of terms in documents in the target language to which the query is being translated to translate query terms. A problem with the parallel corpora approach is that such corpora are rare and building them is prohibitively expensive.